Adaptive Lambda Least-Squares Temporal Difference Learning

Temporal Difference learning or TD(λ) is a fundamental algorithm in the field of reinforcement learning. However, setting TD’s λ parameter, which controls the timescale of TD updates, is generally left up to the practitioner. We formalize the λ selection problem as a bias-variance trade-off where the solution is the value of λ that leads to the smallest Mean Squared Value Error (MSVE). To solve this tradeoff we suggest applying Leave-One-Trajectory-Out CrossValidation (LOTO-CV) to search the space of λ values. Unfortunately, this approach is too computationally expensive for most practical applications. For Least Squares TD (LSTD) we show that LOTO-CV can be implemented efficiently to automatically tune λ and apply function optimization methods to efficiently search the space of λ values. The resulting algorithm, ALLSTDis parameter free and our experiments demonstrate that ALLSTD is significantly computationally faster than the naı̈ve LOTO-CV implementation while achieving similar performance. The problem of policy evaluation is important in industrial applications where accurately measuring the performance of an existing production system can lead to large gains (e.g., recommender systems (Shani and Gunawardana, 2011)). Temporal Difference learning or TD(λ) is a fundamental policy evaluation algorithm derived in the context of Reinforcement Learning (RL). Variants of TD are used in SARSA (Sutton and Barto, 1998), LSPI (Lagoudakis and Parr, 2003), DQN (Mnih et al., 2015), and many other popular RL algorithms. The TD(λ) algorithm estimates the value function for a policy and is parameterized by λ ∈ [0, 1], which averages estimates of the value function over future timesteps. The λ induces a bias-variance trade-off. Even though tuning λ can have significant impact on performance, previous work has generally left the problem of tuning λ up to the practitioner (with the notable exception of (White and White, 2016)). In this paper, we consider the problem of automatically tuning λ in a data-driven way. Defining the Problem: The first step is defining what we mean by the “best” choice for λ. We take the λ value that minimizes the MSVE as the solution to the bias-variance trade-off. Proposed Solution: An intuitive approach is to estimate MSE for a finite set Λ ⊂ [0, 1] and chose the λ ∈ Λ that minimizes an estimate of MSE. Score Values in Λ: We could estimate the MSE with the loss on the training set, but the scores can be misleading due to overfitting. An alternative approach would be to estimate the MSE for each λ ∈ Λ via Cross Validation (CV). In particular, in the supervized learning setting Leave-One-Out (LOO) CV gives an almost unbiased estimate of the loss (Sugiyama et al., 2007). We develop Leave-One-Trajectory-Out (LOTO) CV, but unfortunately LOTO-CV is too computationally expensive for many practical applications. Efficient Cross-Validation: We show how LOTO-CV can be efficiently implemented under the framework of Least Squares TD (LSTD(λ) and Recursive LSTD(λ)). Combining these ideas we propose Adaptive λ Least-Squares Temporal Difference learning (ALLSTD). While a naı̈ve implementation of LOTO-CV requires O(kn) evaluations of LSTD, ALLSTD requires onlyO(k) evaluations, where n is the number of trajectories and k = |Λ|. Our experiments demonstrate that our proposed algorithm is effective at selecting λ to minimize MSE. In addition, the experiments demonstrate that our proposed algorithm is significantly computationally faster than a naı̈ve implementation. Contributions: The main contributions of this work are: 1. Formalize the λ selection problem as finding the λ value that leads to the smallest Mean Squared Value Error (MSVE), 2. Develop LOTO-CV and propose using it to search the space of λ values, 3. Show how LOTO-CV can be implemented efficiently for LSTD, 4. Introduce ALLSTD that is significantly computationally faster than the naı̈ve LOTO-CV implementation, and 5. Prove that ALLSTD converges to the optimal hypothesis. Background Let M = 〈S,A, P, r, γ〉 be a Markov Decision Process (MDP) where S is a countable set of states, A is a finite set of actions, P (s′|s, a) maps each state-action pair (s, a) ∈ S × A to the probability of transitioning to s′ ∈ S in a single timestep, r is an |S| dimensional vector mapping each state s ∈ S to a scalar reward, and γ ∈ [0, 1] is the discount factor. We assume we are given a function φ : S → R ar X iv :1 61 2. 09 46 5v 1 [ cs .L G ] 3 0 D ec 2 01 6 that maps each state to a d-dimensional vector, and we denote by X = φ(S) a d × |S| dimensional matrix with one column for each state s ∈ S. Let π be a stochastic policy and denote by π(a|s) the probability that the policy executes action a ∈ A from state s ∈ S. Given a policy π, we can define the value function